Method and system for material decomposition in dual- or multiple-energy x-ray based imaging

ABSTRACT

A method and system for generating material decomposition images from plural-energy x-ray based imaging, the method comprising: modelling spatial relationships and spectral relationships among the plurality of images by learning features from the plurality of images in combination and one or more of the plurality of images individually with a deep learning neural network; generating one or more basis material images employing the spatial relationships and the spectral relationships; and generating one or more material specific or material decomposition images from the basis material images. The neural network has an encoder-decoder structure and includes a plurality of encoder branches; each of one or more of the plurality of encoder branches encodes two or more images of the plurality of images in combination; and each of one or more of the plurality of encoder branches encodes a respective individual image of the plurality of images.

FIELD OF THE INVENTION

The present invention relates to a deep learning method and system formaterial decomposition in plural—(i.e. dual- or multiple-) energy x-raybased imaging, including cold cathode x-ray CT or radiography,dual-energy CT or radiography, multi-energy CT or radiography, andphoton-counting CT or radiography.

BACKGROUND

Since its introduction, CT has been used widely in the medicaldiagnostic and therapeutic areas. Although CT technology has undergonenumerous advances, its basic principle has been the same: it uses arotating x-ray tube and a row of detectors placed in the gantry tomeasure x-ray attenuations by different tissues inside the body.Compared with other image modalities, CT has many advantages: fastscanning speed, high spatial resolution, and broad availability.Millions of CT examinations are performed annually, making CT one of themost important and widespread imaging modalities used for patient care.

Despite its remarkable success, CT technology has several limitations.One of the most substantial limitations is its low contrast resolution.It cannot reliably differentiate between the material with low inherentcontrast, such as pathologic and healthy tissues. The low contrastresolution is due to the slight difference in x-ray attenuation betweendifferent tissues. For example, it is difficult to reliably assessnoncalcified plaques because the differences in attenuation betweenlipid-rich and lipid-poor noncalcified plaques are minimal. It is alsochallenging to segment soft tissue structures such as cartilage from thekeen CT scans due to the low contrast of the cartilage from thesurrounding soft tissues.

In clinical imaging, contrast agents enhance the material contrast in CTscans. The contrast agents absorb external x-rays, resulting indecreased exposure on the x-ray detector. Contrast agents such asiodinated agents could cause kidney damage and trigger allergicreactions.

In the conventional CT, the attenuation value of each voxel is thecombined attenuation of multiple materials. Dual-energy CT uses twoseparate x-ray photon energy spectra rather than the single energytechnology used in conventional CT. It allows the interrogation ofmaterials that have different attenuation properties at differentenergies. However, due to the limit of two energy bins, the tissuediscrimination is still suboptimal. With more than two energies andnarrow energy ranges, multi-energy CT can concurrently identify multiplematerials with increased accuracy.

Photon-counting CT is an emerging technology that has been showntremendous progress in the last decade. With photon-counting detectors,each photon of the incident x-rays hits the detector element andgenerates an electrical pulse with a height proportional to the energydeposited by the individual photon. Photon-counting CT inherently allowsdual-energy or multi-energy acquisitions at a single source, a singletube, a single acquisition, a single detector, and a single filter.Moreover, the user-defined energy threshold selection allows the choiceof suitable energy thresholds tailored to the specific energy diagnostictask. This task-driven energy-threshold selection helps resolvedifferent tissue types with optimal imaging settings to achieve the bestimage quality or lowest radiation dose.

With either multi-energy CT or photon counting CT, the basic principleof material decomposition is the same: it determines the full energydependence of the attenuation curve in every voxel of a scan. Theassumption is that any human tissue is approximately equivalent to acombination of two or more basis materials, as far as x-ray attenuationproperties are concerned. Although any materials can be employed asbasis materials, water, calcium, iodine or fat are usually used as thebasis materials. Consequently, material decomposition is also referredto as basis material decomposition. The general workflow is as follows.Using multi-energy CT or photon counting CT, the energy selective (orenergy-specific) images are produced by the multi-energy bins. A set ofbasis material images is generated from the energy-selective images.Each basis material image represents the equivalent concentration ofbasis material for each voxel in the scan. The basis images can be usedto obtain images of human tissues such as bone, muscle, and fat througha linear transformation of the basis images. To find the transformationformula for a piece of human tissue, the concentrations of each basismaterial is calculated.

Material decomposition methods have been developed. The simplest methodis inversing the matrix that relates attenuation values to materialconcentrations. Other methods were also advanced, such as optimizationwith regularization. However, with the assumption of the type andnumbers of basis materials, material decomposition is a non-linearill-posed problem and inaccurate decomposition is a problem in currentmethods.

Recently, machine learning, especially deep learning methods, has shownpromise in solving ill-posed problems such as image reconstruction,image resolution enhancement, and voice recognition. In this invention,a deep learning method and system is invented to present the mappingbetween the energy-selective images and material-specific images.

SUMMARY

It is an object of the present invention to provide a method ofgenerating material decomposition images from plural-energy x-ray basedimaging.

According to a first aspect of the invention, there is provided a methodfor generating material decomposition images from a plurality of imagesobtained with plural-energy x-ray based imaging, the plurality of imagescorresponding to respective energies of the plural-energy x-ray basedimaging, the method comprising:

-   -   modelling spatial relationships and spectral relationships among        the plurality of images by learning features from the plurality        of images in combination and one or more of the plurality of        images individually with a deep learning neural network;    -   generating one or more basis material images employing the        spatial relationships and the spectral relationships; and    -   generating one or more material specific or material        decomposition images from the basis material images (such as        through a linear transformation of the basis material images);    -   wherein the neural network has an encoder-decoder structure        (e.g. comprising an encoder network and an decoder network) and        includes a plurality of encoder branches;    -   each of one or more of the plurality of encoder branches encodes        two or more images of the plurality of images in combination        (e.g. together, concatenated or in series); and    -   each of one or more of the plurality of encoder branches encodes        a respective individual image of the plurality of images.

Spatial relationships and spectral relationships are respectivelyrelationships between the spatial information (i.e. of the objects,materials and structures in the images) and spectral information (i.e.the different material attenuations arising from different photonenergies).

It should be noted that the plurality of images obtained withplural-energy x-ray based imaging may be synthetic, in the sense thatthey may not have been obtained simultaneously or in a single scan, butinstead compiled from a plurality of scans.

The one or more encoder branches that encode two or more images of theplurality of images in combination may receive the respective two ormore images in combination, concatenated, etc.., or combine,concatenate, etc.., the respective two or more images before encodingthem.

In an embodiment, each of two or more of the encoder branches encodes arespective different individual image of the plurality of images.

In some embodiments, a first encoder branch encodes a first combinationof two or more images of the plurality of images and a second encoderbranch encodes a second combination of two or more images of theplurality of images, wherein the first combination is different from thesecond combination (though the combinations may include common images).

The plural-energy x-ray based imaging may comprise, for example, coldcathode x-ray radiography, dual-energy radiography, multi-energyradiography, photon-counting radiography, cold cathode x-ray CT,dual-energy CT, multi-energy CT or photon-counting CT.

Advantageously, in some embodiments the encoder branches that encode arespective individual image encode in total all of the images that areencoded in total by the encoder branches that encode two or more images.

However, in some other embodiments, the encoder branches that encode arespective individual image receive in total fewer images (such as byomitting one or more low-energy images) than are encoded in total by theencoder branches that encode two or more images. This may be done, forexample, to reduce computation time.

In still other embodiments, the encoder branches that encode arespective individual image encode in total more images than are encodedin total by the encoder branches that encode two or more images.

The encoder branches that encode a respective individual image mayencode only images than are not encoded by any of the encoder branchesthat encode two or more images. However, more advantageously, theencoder branches that encode a respective individual image encode intotal at least one image than is also encoded by at least one of theencoder branches that encode two or more images.

In one implementation of the invention, the combination of all of theimages (referred to as the ‘energy images’, as each corresponds to arespective x-ray energy bin or energy threshold) is used as input to afirst encoder branch, and each of the individual energy images is usedas the input to a respective one of a plurality of further branches.However, in some implementations, not all of the energy images are usedas input to the first encoder branch and/or as inputs to respectivefurther branches: some energy images may be omitted. For example, if thetargeted basis material images (i.e. those of interest) relate to softtissues only, high energy images may be omitted. On the other hand, highenergy images are useful for differentiating hard materials such asbone, so in implementations in which the basis material images ofinterest relate to hard tissues, low energy images may be omitted.

It may also be advantageous (such as to reduce computing overhead) inthese or other implementations to omit one or more energy images so thatthe neural network is smaller and simpler, with fewer encoder branches.

Hence, in an embodiment, each of the one or more of the encoder branchesrespectively encodes an individual image corresponding to a low x-rayenergy, and the material decomposition images correspond to one or moresoft tissues. In an embodiment, each of the one or more of the encoderbranches respectively encodes an individual image corresponding to ahigh x-ray energy, and the material decomposition images correspond toone or more hard tissues.

It is appreciated that ‘low’ and ‘high’ may be viewed as relative terms,but the appropriate low- or high-energy subset of the entire set ofenergy images can be readily selected by simple experimentation,balancing the quality of the results (measured in terms of resolution orcompleteness of material decomposition) against computing time orcomputing overhead.

However, in one example, the low x-ray energy images (of n-imagesobtained with plural-energy x-ray based imaging) comprise the n-1, n-2or n-3 images of lowest energy. In another example, the low x-ray energyimages comprise the one or two images of lowest energy.

In one example, the high x-ray energy images comprise the n-1, n-2 orn-3 images of highest energy. In another example, the high x-ray energyimages comprise the one or two images of highest energy.

In an embodiment, the deep learning neural network is a trained neuralnetwork, trained with real or simulated training images obtained withreal or simulated plural-energy x-ray based imaging and with basismaterial images. For example, the basis material images may comprise anyone or more (i) HA (hydroxyapatite) images, (ii) calcium images, (iii)water images, (vi) fat images, (v) iodine images, and (vi) muscleimages.

In certain embodiments, the method comprises generating any one or moreof (i) a bone marrow decomposition image, (ii) a knee cartilagedecomposition image, (iii) an iodine contrast decomposition image, (iv)a tumor decomposition image, (v) a muscle and fat decomposition image,(vi) a metal artefact reduction image, and (vii) a beam hardeningreduction image.

The method may comprise:

-   -   generating one or more bone marrow images, and diagnosing,        identifying or monitoring bone marrow related disease using the        one or more bone marrow images;    -   generating one or more knee cartilage images, and diagnosing,        identifying or monitoring osteoarthritis or rheumatoid arthritis        using the one or more bone marrow images;    -   generating one or more iodine contrast image, and diagnosing,        identifying or monitoring a tumor; and/or    -   generating one or more muscle images, and diagnosing,        identifying or monitoring sarcopenia.

The method may comprise:

-   -   generating any one or more (a) bone marrow images, (b) knee        cartilage images, (c) iodine contrast images, and (d) muscle        images;    -   generating one or more metal artefact images and/or one or more        beam hardening reduction images; and    -   improving image quality of the bone marrow, knee cartilage,        iodine contrast and/or muscle images using the metal artefact        and/or beam hardening reduction images.

The method may include training or retraining deep learning models usingthe neural network.

The method may include combining features extracted by the one or moreencoder branches that encode two or more images in combination andfeatures extracted by the one or more encoder branches that encoderespective individual images using a concatenation layer at the end ofor after an encoder network of the neural network.

In other embodiments, the method includes combining features extractedby the one or more encoder branches that encode two or more images incombination and features extracted by the one or more encoder branchesthat encode respective individual images using one or more concatenationoperations at plural levels of an encoder network of the neural network.

In still other embodiments, the method includes combining featuresextracted by the one or more encoder branches that encode two or moreimages in combination and features extracted by the one or more encoderbranches that encode respective individual images using concatenationoperations that connect an encoder network of the neural network and andecoder network of the neural network at multiple levels.

In yet other embodiments, the method includes combining featuresextracted by the one or more encoder branches that encode two or moreimages in combination and features extracted by the one or more encoderbranches that encode respective individual images, but an encodernetwork of the neural network and an decoder network of the neuralnetwork are not connected at multiple levels.

According to this aspect, there is also provided a materialdecomposition image, generated according to the method of this aspect(including any of its embodiments) from a plurality of images obtainedwith plural-energy x-ray based imaging.

According to a second aspect of the invention, there is provided asystem for generating material decomposition images from a plurality ofimages obtained with plural-energy x-ray based imaging, the plurality ofimages corresponding to respective energies of the plural-energy x-raybased imaging, the system comprising:

-   -   the neural network has an encoder-decoder structure (e.g.        comprising an encoder network and an decoder network) and        includes a plurality of encoder branches;    -   wherein each of one or more of the plurality of encoder branches        is configured to encode two or more images of the plurality of        images in combination (e.g. together, concatenated or in        series); and    -   each of one or more of the plurality of encoder branches is        configured to encode a respective individual image of the        plurality of images as input;    -   the neural network is configured to model spatial relationships        and spectral relationships among the plurality of images by        learning features from the plurality of images in combination        and one or more of the plurality of images individually with a        deep learning neural network, and to generate one or more basis        material images employing the spatial relationships and the        spectral relationships; and    -   the system is configured to generate one or more material        specific or material decomposition images from the basis        material images.

The one or more encoder branches that encode two or more images of theplurality of images in combination may receive the respective two ormore images in combination, concatenated, etc., or combine, concatenate,etc., the respective two or more images before encoding them.

In an embodiment, each of two or more of the encoder branches isconfigured to encode a respective different image of the plurality ofimages.

In some embodiments, a first encoder branch is configured to encode afirst combination of two or more images of the plurality of images asinput and a second encoder branch is configured to encode a secondcombination of two or more images of the plurality of images as input,wherein the first combination is different from the second combination(though the combinations may include common images).

The plural-energy x-ray based imaging may comprise cold cathode x-rayradiography, dual-energy radiography, multi-energy radiography,photon-counting radiography, cold cathode x-ray CT, dual-energy CT,multi-energy CT or photon-counting CT.

Advantageously, in some embodiments the encoder branches configured toencode a respective individual image receive in total all of the imagesthat are encoded in total by the encoder branches configured to encodetwo or more images.

However, in other embodiments, the encoder branches that encode arespective individual image are configured to encode in total fewerimages (such as by omitting one or more low-energy images) than areencoded in total by the encoder branches that encode two or more images(such as to reduce computation time).

In still other embodiments, the encoder branches that encode arespective individual image are configured to encode in total moreimages than are encoded in total by the encoder branches that encode twoor more images.

The encoder branches that encode a respective individual image mayencode only images than are not encoded by any of the encoder branchesthat encode two or more images. However, more advantageously, theencoder branches that encode a respective individual image encode intotal at least one image than is also encoded by at least one of theencoder branches that encode two or more images.

The deep learning neural network may be a trained neural network,trained with real or simulated training images obtained with real orsimulated plural-energy x-ray based imaging and with basis materialimages. For example, the basis material images may comprise any one ormore (i) HA (hydroxyapatite) images, (ii) calcium images, (iii) waterimages, (vi) fat images, (v) iodine images, and (iv) muscle images.

The system may be configured to generate any one or more of (i) a bonemarrow decomposition image, (ii) a knee cartilage decomposition image,(iii) an iodine contrast decomposition image, (iv) a tumor decompositionimage, (v) a muscle and fat decomposition image, (vi) a metal artefactreduction image, and (vii) a beam hardening reduction image.

In an embodiment, the system is configured:

-   -   to generate one or more bone marrow images, and to diagnose,        identify or monitor bone marrow related disease using the one or        more bone marrow images;    -   to generate one or more knee cartilage images, and to diagnose,        identify or monitor osteoarthritis or rheumatoid arthritis using        the one or more bone marrow images;    -   to generate one or more iodine contrast image, and to diagnose,        identify or monitor a tumor; and/or    -   to generate one or more muscle images, and to diagnose, identify        or monitor sarcopenia. The system may be configured to:    -   generate any one or more (a) bone marrow images, (b) knee        cartilage images, (c) iodine contrast images, and (d) muscle        images;    -   generate one or more metal artefact images and/or one or more        beam hardening reduction images; and    -   improve image quality of the bone marrow, knee cartilage, iodine        contrast and/or muscle images using the metal artefact and/or        beam hardening reduction images.

The system may include deep learning model trainer configured to trainor retrain deep learning models using the neural network.

The system may be configured to combine features extracted by the one ormore encoder branches that encode two or more images in combination andfeatures extracted by the one or more encoder branches that encoderespective individual images using a concatenation layer at the end ofor after an encoder network of the neural network.

In other embodiments, the system may be configured to combine featuresextracted by the one or more encoder branches that encode two or moreimages in combination and features extracted by the one or more encoderbranches that encode respective individual images using one or moreconcatenation operations at plural levels of an encoder network of theneural network.

In still other embodiments, the system may be configured to combinefeatures extracted by the one or more encoder branches that encode twoor more images in combination and features extracted by the one or moreencoder branches that encode respective individual images usingconcatenation operations that connect an encoder network of the neuralnetwork and an decoder network of the neural network at multiple levels.

In yet other embodiments, the system may be configured to combinefeatures extracted by the one or more encoder branches that encode twoor more images in combination and features extracted by the one or moreencoder branches that encode respective individual images, wherein anencoder network of the neural network and an decoder network of theneural network are not connected at multiple levels.

According to a third aspect of the invention, there is provided acomputer program comprising program code configured, when executed byone of more computing devices, to implemented the method of the firstaspect (and any of its embodiments). According to this aspect, there isalso provided a computer-readable medium (which may be non-transient),comprising such a computer program.

It should be noted that any of the various individual features of eachof the above aspects of the invention, and any of the various individualfeatures of the embodiments described herein, including in the claims,can be combined as suitable and desired.

DRAWINGS

In order that the invention may be more clearly ascertained, embodimentswill now be described by way of example with reference to the followingdrawing, in which:

FIG. 1 is a schematic view of an image processing system according to anembodiment of the present invention.

FIG. 2 is a schematic flow diagram of the general workflow of the systemof FIG. 1 .

FIG. 3A is a schematic view of a deep learning neural network forgenerating material decomposition images from plural-energy imagesaccording to an embodiment of the present invention.

FIG. 3B is a schematic view of a deep learning neural network forgenerating material decomposition images from plural-energy imagesaccording to another embodiment of the present invention.

FIG. 4 is a schematic view of a deep learning neural network forgenerating material decomposition images from plural-energy imagesaccording to an embodiment of the present invention.

FIG. 5 is a schematic view of the training of the deep learning model ormodels by the deep learning model trainer of the system of FIG. 1 .

FIGS. 6A and 6B are schematic views of the preparation of exemplarytraining data.

FIG. 7 illustrates an exemplary operation of the system of FIG. 1 .

DETAILED DESCRIPTION

FIG. 1 is a schematic view of an image processing system 10 (ofapplication in particular for processing medical images) according to anembodiment of the present invention.

System 10 includes an image processing controller 12 and a userinterface 14 (including a GUI 16). User interface 14 includes one ormore displays (on one or more of which may be generated GUI 16), akeyboard and a mouse, and optionally a printer.

Image processing controller 12 includes at least one processor 18 and amemory 20. Instructions and data to control operation of processor 18are stored in memory 20.

System 10 may be implemented as, for example, a combination of softwareand hardware on a computer (such as a server, personal computer ormobile computing device) or as a dedicated image processing system.System may optionally be distributed; for example, some or all of thecomponents of memory 20 may be located remotely from processor 18; userinterface 14 may be located remotely from memory 20 and/or fromprocessor 18 and, indeed, may comprise a web browser or a mobile deviceapplication.

Memory 20 is in data communication with processor 18, and typicallycomprises both volatile and non-volatile memory (and may include morethan one of type of memory), including RAM (Random Access Memory), ROMand one or more mass storage devices.

As is discussed in greater detail below, processor 18 includes an imagedata processor 30, which includes a basis material image generator 32, adiagnostic/monitoring task image generator 34 (including a decomposer36), and an additional task-driven image generator 38. Processor 18further includes a deep learning model trainer 40 (which includes one ormore deep learning neural networks 42), an I/O interface 44 and anoutput in the form of a results output 46. Deep learning model trainer40 may be omitted in some implementations of this and other embodiments,as it is required only if system 10 is itself to train deep learningmodel(s) 58, rather than access one or more suitable deep learningmodels from an external source.

Memory 20 includes program code 50, image data store 52, non-image datastore 54, training data store 56, trained deep learning model(s) 58,generated basis material image store 60 and generated material specificor material decomposition image store 62. Image processing controller isimplemented, at least in part, by processor 18 executing program code 50from memory 20.

In broad terms, the I/O interface 44 is configured to read or receiveimage data (such as in DICOM format) and non-image data, pertainingto—for example—subjects or patients, into image data store 52 andnon-image data store 54 of memory 20, respectively, for processing. Thenon-image data stored in non-image data store 54 comprises broadinformation such as energies, desired materials and desired tasks, andis accessible by image generators 32, 34, 36 for use in imagegeneration.

Basis material image generator 32 of image data processor 12 generatesone or more sets of basis material images with one or more machinelearning models (drawn from deep learning model(s) 58).Diagnostic/monitoring task image generator 34 uses decomposer 36 togenerate one or more sets of material specific or material decompositionimages (suitable for, for example, diagnostic or monitoring tasks) usingthe basis material images, and additional task-driven image generator 38generates at least one further set of images (such as beam hardening ormetal artefact reduced images). I/O interface 44 outputs the results ofthe processing to, for example, results output 46 and/or to GUI 16.

System 10 employs one or more deep learning models to accurately andreproducibly generate the basis material images. The basis materialimages are then used for generating images of different tissues andmaterials, especially of low contrast tissues and materials, which canin turn be used in pathology or disease identification and monitoring(such as of disease progression). For example, cartilage segment imagesfrom the knee scan may be used for osteoarthritis or rheumatoidarthritis diagnosis and/or monitoring; bone marrow segment images frommusculoskeletal scans may be used for related diseases diagnosis andmonitoring of associated diseases or pathologies; pathological andnormal tissue images from a scan of a patient may be used for diagnosisand monitoring of a tumor; simultaneous material decomposition ofmultiple contrast agents from a CT scan may be used for the diagnosis oridentification, and staging, of renal abnormalities; muscle extractedimages may be used for sarcopenia diagnosis and/or monitoring.

System 10 can also generate images for other tasks (using additionaltask-driven image generator 38). For example, system 10 can generatebeam-hardening or metal artefact reduced images based on theaforementioned basis material images for better image quality. Beamhardening or metal artefact effects occur when a polychromatic x-raybeam passes through an object, resulting in selectiveattenuation—principally affecting lower energy photons. As a result,higher energy photons solely or excessively contribute to the beam,thereby increasing the mean beam energy—an effect known as ‘beamhardening.’ As the full energy-dependent attenuation is considered inmaterial decomposition, it is thus desirable that the decomposed imagesbe free of beam-hardening and metal artefact effects.

Thus, referring to FIG. 1 , system 10 is configured to receive two typesof data pertaining to a subject or patient: image data and non-imagedata. The image data is on the form of plural-energy images based on orderived from x-ray imaging, such as may be generated in cold cathodex-ray radiography, dual-energy CT, multi-energy CT or photon-countingCT. The non-image data includes information about the plural-energyimages, such as the energies at which the plural-energy images weregenerated, information about the desired basis material images, such asthe type and number of the basis material images, and information aboutthe desired analysis or analyses, such as diseasediagnosis/identification/monitoring or additional tasks (e.g., beamhardening or metal artefact reduction). System 10 stores image data andnon-image data in the image data store 52 and non-image data store 54,respectively.

As mentioned above, image data processor 30 includes three components:basis material image generator 32, diagnostic/monitoring task imagegenerator 34, and additional task-driven image generator 38. The imagedata and non-image data are received by image data processor 30 frommemory 20. Based on the plural-energy images and the basis material inthe non-image data, image data processor 30 selects one or more suitabledeep learning models 58 to generate one or more sets of basis materialimages. Based on the task information, image data processor 30 generatesimages (e.g., human tissues, contrast agents images) for diseasediagnosis/identification and/or monitoring, and images (e.g. beamhardening and metal artefact reduced images) for better image quality.

Deep learning model trainer 40 pre-trains deep learning models 58 usingtraining data (from training data store 56) that includes labels orannotations that constitute the ground truth for machine learning. Thetraining data is prepared so as to be suitable for training adeep-learning model for generating basis material images from theplural-energy images. The training data consists of both knownplural-energy images and known basis material images. The labelsindicate the energy bin of each energy image (that is, an imagecorresponding to a particular energy threshold or bin) and the materialinformation (e.g., material name and material density) of the basismaterial images. The training data can be in the form of real clinicaldata, real phantom data, simulated data, or a mixture of two or more ofthese.

As mentioned above, deep learning model trainer 40 is configured totrain one or more deep learning models (and to retrain or update traindeep learning models) using neural network 42 and the training data, butin other embodiments machine learning model trainers may be configuredor used only to retrain or update (i.e., re-train) one or more existingdeep learning models.

Image data processor 30 selects one or more suitable deep learningmodels from deep learning model(s) 58, based on the plural-energy imagesand the targeted basis material(s) (as identified in the non-imagedata). Basis material image generator 32 generates images of thetargeted basis material. Diagnostic/monitoring task image generator 34generates images according to the information concerningdiagnosis/identification and/or monitoring tasks (as also identified inthe non-image data), from the generated basis material images.Optionally, additional task-driven image generator 38 generates imagesaccording to the information of the additional tasks (as also identifiedin the non-image data), from the generated basis material images.

The basis material images, diagnostic/monitoring images, and/oradditional task-driven images are outputted to user interface 14 viaresults output 46 and I/O interface 44.

FIG. 2 is a flow diagram 70 of the general workflow of system 10 of FIG.1 . Referring to FIG. 2 , at step 72 system 10 receives plural-energyimages (generated by, for example, dual-energy, multi-energy orphoton-counting CT or radiography) and reads the images into image datastore 52. At step 74, system 10 receives associated non-image data andreads that data into non-image data store 54.

Memory 20 is advantageously configured to allow high-speed access ofdata by system 10. For example, if system 10 is implemented as acombination of software and hardware on a computer, the images aredesirably read into RAM of memory 20.

At step 76, image data processor 30 selects one or more suitable deeplearning models from the trained deep learning model(s) 58. The deeplearning model selection is based on the energy informationcharacterizing the plural-energy images and the information concerningthe targeted basis material, both contained in the non-image data. Anyparticular model is trained using the images of specific energies togenerate a specific set of basis material images; hence, more than onesuitable model may be trained and available. According to the energiesand desired basic material specs, the corresponding model or models areis selected. If plural models are selected, they are used in parallel.

For example, one deep learning model may be selected for use with allloaded images for generating one set of basis material images. Inanother example, more than one deep learning model is chosen for usewith all loaded images for generating several sets of basis materialimages. In another example, more than one deep learning model isselected to use with respective subsets of the loaded images, forgenerating one or more sets of basis material images.

The selected deep learning model or models include spatial relationshipsand spectral relationships learned from training data. At step 78, basismaterial images generator 32 generates the basis material images fromthe loaded subject or patient images in image data store 52 using theone or more selected deep learning models and these spatial and spectralrelationships, and saves the generated basis material images ingenerated basis material image store 60.

At step 80, diagnostic/monitoring task image generator 34 uses thegenerated basis material images to decompose the original subject orpatient images in image data store 52 and thereby generate materialspecific or material decomposition images of, in this example, specific,different (e.g. human) tissues, suitable for disease identification,diagnosis and/or monitoring, and saves these material specific ordecomposition images in generated material specific or materialdecomposition image store 62.

At step 82, image data processor 30 determines whether—according to theassociated non-image data 54 indicating the desired task(s)—additionaltask-driven image generator 38 is required to generate any images. Ifnot, processing ends. If so, at step 84, additional task-driven imagegenerator 38 generates the appropriate task-driven images, such as beamhardening reduced images and/or metal artefact reduced images.Processing then ends.

FIG. 3A is a schematic view of a deep learning neural network 90 (suchas may be employed as neural network 42 of system 10), for generatingmaterial decomposition images from plural-energy images according to anembodiment of the present invention. Neural network 90 is shown withinput in the form of n plural-energy x-ray based images 92 (where n≥2)and output in the form of basis material images 94. Neural network 90 isconfigured to generate basis material images 94 from the images 92. Thatis, the functional mapping between the input images 92 and output basismaterial images 94 is approximated by neural network 90, which isconfigured to predict material-specific images using the images 92 asinput. Material decomposition images can then be generated from thegenerated basis material images 94.

Neural network 90 comprises an encoder network 96 and a decoder network98. Encoder network 96 encrypts the structures of the input images (e.g.some or all of images 92) into a feature representation at multipledifferent levels. Decoder network 98 projects the discriminative featurerepresentation learnt by encoder network 96 into the pixel/voxel spaceto get a dense classification. In one example, the encoding performed byencoder network 96 includes convolution operations and down-samplingoperations; the decoding performed by decoder network 98 includesconvolution operations and up-sampling operations. In another example,the encoding performed by encoder network 96 and/or the decodingperformed by decoder network 98 include concatenation operations.

Encoder network 96 has a plural-branch structure, with a first set 100 ₁and a second set 100 ₂ of encoder branches (each set having one or moreencoder branches). Each of the branches of the first set 100 ₁ ofencoder branches encodes a plurality of images selected from images 92(which may comprise all of images 92) in concatenated form. (It shouldbe noted that this or these pluralities of images selected from images92 for processing in concatenated form may be inputted either inconcatenated form or non-concatenated form. In the latter case, theencoder network first concatenates the images.)

Each of the branches of the second set 100 ₂ of encoder branches encodesan individual image selected from images 92. First set 100 ₁ and secondset 100 ₂ may include, in total, the same or different numbers ofimages.

In the example of FIG. 3A, first set 100 ₁ of encoder branches includes,in this example, one encoder network branch 96 ₀ (comprising ‘Encodernetwork 0’) for encoding a plurality of images 92 (in this example allof images 92) in concatenated form. Second set 100 ₂ of encoder branchesincludes a plurality m of encoder network branches 96 ₁, 96 ₂, . . . ,96 _(m) (comprising respectively ‘Encoder network 1’, ‘Encoder network2’, . . . , ‘Encoder network m’) for encoding each of the respective,individual input images 92 ₁, 92 ₂, . . . , 92 _(m), where m≤n. (Notethat images 1, 2, ... , m need not be sequential or comprise the first mimages of images 92. Also, encoder network branch 96 ₀ may be configuredto receive a plurality—but not all—of the images 92 in concatenatedform.) The individual images 92 ₁, 92 ₂, . . . , 92 _(m) are generallyof a conventional format for the respective imaging modality (e.g.DICOM, JPEG, TIFF or other imaging files), so are typically two- orthree-dimensional images comprising pixels or voxels but, as they havean extra dimension indicative of energy threshold or bin, could bedescribed as three- or four-dimensional. Likewise, the images 92 inconcatenated or combined form have an extra dimension (indicative ofenergy threshold or bin), so may also be described as typically three-or four-dimensional.

Encoder network branch 96 ₀ of the first set 100 ₁ learns relationshipsamong images 92 inputted into that branch and effectively combines them.Encoder network branch 96 ₀ of the second set 100 ₂ learn the featuresof each individual image 92 ₁, 92 ₂, . . . , 92 _(m) independently. Thefeature representations learned by the first set 100 ₁ of networkbranches (viz. network branch 96 ₀) and by the second set 100 ₂ ofnetwork branches 96 ₁, 96 ₂, . . . , 96 _(m) are combined as the inputof decoder network 98.

In one example, the features extracted by first set 100 ₁ of encodernetwork branches 96 ₀ and by second set 100 ₂ of encoder networkbranches 96 ₁, 96 ₂, . . . , 96 _(m) are combined using a concatenationlayer (not shown) at the end of or after encoder network 96. In anotherexample (cf. the embodiment in FIG. 4 ), the features extracted fromfirst set 100 ₁ of branches and second set 100 ₂ of branches arecombined using one or more concatenation operations at the plural levelsof encoder network 96.

In a further example (cf. the embodiment in FIG. 4 ), concatenationoperations connect the encoder network 96 and decoder network 98 atmultiple levels. In still another example, encoder network 96 is notconnected to decoder network 98 at multiple levels.

As mentioned above, all of images 92 may be concatenated to form theinput (or concatenated image) for input into first branch 96 ₀;alternatively, only some (but a plurality) of the input images 92 ₁, 92₂, . . . , 92 _(m) may be concatenated to form the input (orconcatenated image) for input into first set 100 ₁ of encoder branches(viz. encoder branch 96 ₀). In one example, all of the images 92 areseparately input into second set 100 ₂ of encoder branches but, inanother example, some (i.e. one or more) of the images 92 ₁, 92 ₂, . . ., 92 _(n) might not be encoded by second set 100 ₂ of encoder branches.In addition, it should be noted that the images that are input into thefirst and second sets 100 ₁, 100 ₂ of encoder branches need not be thesame, but are drawn from the same multi-energy images 92.

Thus, deep learning neural network 90, which may thus be described as amulti-branch encoder-decoder deep learning network, generates the basismaterial images 94 by inherently modelling spatial and spectralrelationships among the plural-energy images 92.

FIG. 3B is a schematic view of a deep learning neural network 90′ (suchas may be employed as neural network 42 of system 10), which iscomparable to neural network 90 of FIG. 3A so like numerals have beenused to indicate like features. Neural network 90′ is thus also adaptedfor generating material decomposition images from plural-energy imagesaccording to an embodiment of the present invention.

Neural network 90′ includes an encoder network 96′ that includes firstand second sets 100 ₁′, 100 ₂′ of encoder branches. Neural network 90′differs from neural network 90 of FIG. 3A in that the first set 100 ₁′of encoder branches of neural network 90′ includes at least two encoderbranches 96 ₀′, 96 ₁′ comprising encoder network 0′ and encoder network1′, respectively, each of which is configured to receive a plurality ofconcatenated images (respectively images 102 ₁ and images 102 ₂)selected from images 92.

Images 102 ₁ and images 102 ₂ may comprise the same or a differentnumbers of images and, in either case, may constitute overlapping ornon-overlapping sets of images.

FIG. 4 is a schematic view of a deep learning neural network 110 (suchas may be employed as neural network 42 of system 10), for generatingbasis material images from plural-energy images according to anembodiment of the present invention. Neural network 110 is shown withinput in the form of a plurality (in this example, four) ofplural-energy x-ray based images 112.

Neural network 110 includes a multi-branch encoder network 114 and adecoder network 116. In this embodiment, encoder network 114 has a firstset of encoder branches comprising a single branch: a first branch 118that receives the combination of all four images 112 as input. Encodernetwork 114 has a second set of encoder branches comprising, in thisexample, two branches: a second branch 122 that receives the first image112 ₁ (being the first of plural-energy x-ray based images 112) asinput, and a third branch 126 that receives the third image 112 ₃ (beingthe third of plural-energy x-ray based images 112) as input.

The encoder network structure of each of the three encoder branches 118,122, 126 is identical, each encoder branch containing three stagesdefined by the size of its feature maps, with each stage containing theconvolutions, batch normalization, and ReLU (Rectified Linear Unit)functions or operations. Thus, the first branch 118 comprises a firststage 118 ₁ that includes 16 channel first feature map 120 ₁, which isthe same width and height as the original combination of images 112(which may also be regarded as a part of first stage 118 ₁ of the firstbranch). The second stage 118 ₂ includes 16 channel second feature map120 ₂ and 64 channel third feature map 120 ₃, while the third stage 118₃ includes 64 channel fourth feature map 120 ₄ and 128 channel fifthfeature map 120 ₅.

Likewise, the second branch 122 comprises a first stage 122 ₁ thatincludes 16 channel first feature map 124 ₁, which is the same width andheight as the first individual image 112 ₁ (which may also be regardedas a part of first stage 122 ₁ of the second branch). The second stage122 ₂ includes 16 channel second feature map 124 ₂ and 64 channel thirdfeature map 124 ₃, while the third stage 122 ₃ includes 64 channelfourth feature map 124 ₄ and 128 channel fifth feature map 124 ₅.

The third branch 126 comprises a first stage 126 ₁ that includes 16channel first feature map 128 ₁, which is the same width and height asthird individual image 112 ₃ (which may also be regarded as a part ofthe first stage 126 ₁ of the third branch). The second stage 126 ₂includes 16 channel second feature map 128 ₂ and 64 channel thirdfeature map 128 ₃, while the third stage 126 ₃ includes 64 channelfourth feature map 128 ₄ and 128 channel fifth feature map 128 ₅. Thefeature map 120 ₁, 124 ₁, 128 ₁ of each respective first stage 118 ₁,122 ₁, 126 ₁, and the last feature map 120 ₃, 124 ₃, 128 ₃ of eachrespective second stage 118 ₂, 122 ₂, 126 ₃ undergoes max pooling,reducing the size of the feature maps and allowing encoder network 114to find the global features of the respective input images 112, 112 ₁,112 ₃. (Note that the pooling operation is on the featurerepresentations or maps between two stages, which is why there are twopooling operations for the three stages.)

In this embodiment, decoder network 116 also contains three stages, withthe last feature map 120 ₅ of the first stage of decoder network 114also acting as the first stage of encoder network 116. Each of the threestages 130 ₁, 130 ₂, 130 ₃ of decoder network 116 is defined by the sizeof its respective feature maps: first stage 130 ₁ includes 128 channelfeature map 120 ₅, second stage 130 ₂ includes 64 channel feature map132 ₁ and 32 channel feature map 132 ₂, and third stage 130 ₃ includes16 channel feature map 132 ₃ and 4 channel feature map 134 (the latterbeing the outputted basis material image(s)). Each of these three stages130 ₁, 130 ₂, 130 ₃ contains convolutions, batch normalization, and ReLUoperations, and the feature maps of stages 130 ₁, 130 ₂, 130 ₃ undergoaverage pooling (i.e. a pooling operation is applied to the feature mapsbetween stages 130 ₁ and 130 ₂, and between stages 130 ₂ and 130 ₃),bringing the feature map dimensions back to match those of input images112, 112 ₁, 112 ₃.

In this embodiment, the feature maps of each stage of the three branches118, 122, 126 of encoder network 114 are concatenated (hence,respectively, feature maps 120 ₁, 124 ₁, 128 ₁; feature maps 120 ₃, 124₃, 128 ₃; and feature maps 120 ₅, 124 ₅, 128 ₅), and then concatenatedwith the feature maps at the corresponding stage of decoder network 116(hence, respectively, feature maps 120 ₅, 132 ₁ and 130 ₃). Theconnection at the multiple levels between multi-branch encoder network114 and decoder 116 enables neural network 110 to learn the localdetails of input images 112, 112 ₁, 112 ₃.

FIG. 5 is a flow diagram 140 of the training—by deep learning modeltrainer 40—of the deep learning model or models stored ultimately indeep learning model(s) 58. At step 142, training data are prepared orsourced, the training data comprising plural-energy x-ray based imagesand basis material images. The training data may be real data, simulateddata, or combinations thereof. In one example, the training data isgenerated using phantoms of different known materials. In anotherexample, the training data is simulated based on the knowncharacteristics of known materials under different x-ray energies. Insome examples, the training data comprises real data only or simulateddata only. In another example, the training data comprises some realdata and some simulated data. Hence, preparing the training data mayinvolve—for example—generating (or sourcing) real training data (seestep 144 a) and/or simulating (or sourcing simulated) training data (seestep 144 b).

The process optionally includes step 146, where the training data isincreased using a data augmentation method. This may entail the additionof Gaussian noise to the training data to improve the robustness of themodel training, and/or dividing the training data into patches toincrease the quantity of training data.

At step 148, the training data are labelled with the appropriate,correct labels. Each ‘energy image’ (that is, an individual imagecorresponding to a single energy threshold or energy bin) is labelledwith the relevant energy threshold or energy bin (see step 150 a), andeach basis material image is labelled with the relevant material (seestep 150 b). At step 152, deep learning model trainer 40 trains one ormore deep learning models, employing the correctly labelled energyimages and basis material images. Step 152 may entail updating (orretraining) one or more trained deep learning models, if such modelshave previously been trained and the training data prepared or sourcedat step 142 is new or additional training data.

At step 154, the trained or retrained model or models are deployed foruse, by being stored in machine learning model(s) 58. Processing thenends, unless the process includes optional step 156 at which deeplearning model trainer 40 determines whether retraining or furthertraining is to be conducted. If not, processing ends, but if deeplearning model trainer 40 determines that retraining or further trainingis to be conducted, processing returns to step 142.

In use, system 10 inputs one or more plural-energy x-ray based imagesinto one or more of the now trained deep learning models 58, whichprocess the images and outputs a set of basis material images.

FIGS. 6A and 6B are schematic views of exemplary training datapreparation techniques. FIG. 6A shows the preparation 160 of trainingdata using real data. For example, one or more phantoms are used, eachhaving an insert that contains a known material (e.g. HA(hydroxyapatite), iodine, calcium, blood or fat) or a mixture (e.g.iodine and blood) of some or all of the known materials.

The composition, concentration, size and location of each materialinsert are known. The phantom is scanned 162 using, for example, coldcathode x-ray radiography, dual-energy CT, multi-energy CT orphoton-counting CT, such that the plural-energy images are generated 164with two or more energy thresholds or energy bins. In this example, theaim is to generate three basis material-specific images: a HA image, aniodine image and a fat image. Each basis material-specific image is thusgenerated 166 with the known concentration, size and location of eachmaterial insert.

FIG. 6B shows the preparation 170 of training data using simulated data.For example, one or more phantoms are simulated 172, again with insertscontaining known materials (e.g., iodine, calcium, blood, and fat) and amixture of some of the known materials. The concentration, size andlocation of each material insert are known. The plural-energy images aresimulated 174 based on the known materials and specific energies, suchas by creating those images by referring to the real scans, acquired bya plural-energy or photon-counting CT, etc., but with a differentconcentration of the material. For example, real scans may be acquiredby scanning a real phantom with inserts comprising iodine withrespective iodine concentrations of 2, 8 and 16 mg/cc using aphoton-counting CT. Simulated scans of 20, 25, 30 mg/cc iodine insertscan then be generated by applying a linear fitting to the CT numbers ofthe real scans, as photon-counting CT maintains a strong linearrelationship between CT numbers and the concentrations of the samematerial. In another example, the energy images are created bymathematical simulation based on the known reaction of known materialunder different x-ray energies. The basis material-specific images aresimulated 176 with the concentration, size and location of each materialinsert.

FIG. 7 illustrates an exemplary work flow 180 of system 10 of FIG. 1 ,for the particular case of a patient that has been injected with aniodine contrast agent then scanned 182 using photon-counting CT. Fiveimages are generated using respectively the five energy thresholds (thatis, the energies above which x-rays are counted in the fivecorresponding energy bins), thereby producing a 25 keV threshold image184, a 35 keV threshold image 186, a 45 keV threshold image 188, a 55keV threshold image 190 and a 65 keV threshold image 192. From theseimages 184, 186, 188, 190, 192, the trained deep learning model ormodels generate 194 four basis material images: a calcium image 196, awater image 198, a fat image 200 and an iodine image 202. From differentlinear combinations of the basis material images 196, 198, 200, 202,various functional images are generated for diseasediagnostic/monitoring and/or other tasks. For example:

-   -   i) a bone marrow image 204 may be generated for bone marrow        related disease diagnostic/monitoring (using, for example, HA        (hydroxyapatite)+Fat+Water+soft tissue as basis        materials/images);    -   ii) a knee cartilage image 206 may be generated for        osteoarthritis or rheumatoid arthritis diagnostic/monitoring        (using, for example, HA+Fat+Water+soft tissue as basis        materials/images);    -   iii) an iodine contrast image 208 may be generated for tumor        diagnostic/monitoring (using, for example, HA+Fat+Water+soft        issue+iodine as basis materials/images); and    -   iv) a metal artefact and beam hardening reduction image 210 may        be generated for better image quality (using, for example,        HA+soft tissue as basis materials/images).

It will be understood by persons skilled in the art of the inventionthat many modifications may be made without departing from the scope ofthe invention. In particular it will be apparent that certain featuresof embodiments of the invention can be employed to form furtherembodiments.

It is to be understood that, if any prior art is referred to herein,such reference does not constitute an admission that the prior art formsa part of the common general knowledge in the art in any country.

In the claims that follow and in the preceding description of theinvention, except where the context requires otherwise due to expresslanguage or necessary implication, the word “comprise” or variationssuch as “comprises” or “comprising” is used in an inclusive sense, i.e.to specify the presence of the stated features but not to preclude thepresence or addition of further features in various embodiments of theinvention.

The invention claimed is:
 1. A method for generating materialdecomposition images from a plurality of images obtained withplural-energy x-ray based imaging, the plurality of images correspondingto respective energies of the plural-energy x-ray based imaging, themethod comprising: modelling spatial relationships and spectralrelationships among the plurality of images by learning features fromthe plurality of images in combination and one or more of the pluralityof images individually with a deep learning neural network; generatingone or more basis material images employing the spatial relationshipsand the spectral relationships; and generating one or more materialspecific or material decomposition images from the basis materialimages; wherein the neural network has an encoder-decoder structure andincludes a plurality of encoder branches; each of one or more of theplurality of encoder branches encodes two or more images of theplurality of images in combination; and each of one or more of theplurality of encoder branches encodes a respective individual image ofthe plurality of images.
 2. The method as claimed in claim 1, wherein:i) each of two or more of the encoder branches encodes a respectivedifferent individual image of the plurality of images as input; and/orii) a first encoder branch encodes a first combination of two or moreimages of the plurality of images and a second encoder branch encodes asecond combination of two or more images of the plurality of images,wherein the first combination is different from the second combination.3. The method as claimed either in claim 1, wherein the plural-energyx-ray based imaging comprises cold cathode x-ray radiography,dual-energy radiography, multi-energy radiography, photon-countingradiography, cold cathode x-ray CT, dual-energy CT, multi-energy CT orphoton-counting CT.
 4. The method as claimed in claim 1, wherein: a) theencoder branches that encode a respective individual image receive intotal all of the images that are received in total by the encoderbranches that encode two or more images; or b) the encoder branches thatencode a respective individual image encode in total fewer images thanare encoded in total by the encoder branches that encode two or moreimages; or c) the encoder branches that encode a respective individualimage encode in total more images than are encoded in total by theencoder branches that encode two or more images.
 5. The method asclaimed in claim 1, wherein the deep learning neural network is atrained neural network, trained with real or simulated training imagesobtained with real or simulated plural-energy x-ray based imaging andwith basis material images.
 6. The method as claimed in claim 5, whereinthe basis material images comprise any one or more (i) hydroxyapatiteimages, (ii) calcium images, (iii) water images, (vi) fat images, (v)iodine images, and (vi) muscle images.
 7. The method as claimed in claim1, comprising generating any one or more of (i) a bone marrowdecomposition image, (ii) a knee cartilage decomposition image, (iii) aniodine contrast decomposition image, (iv) a tumor decomposition image,(v) a muscle and fat decomposition image, (vi) a metal artefactreduction image, and (vii) a beam hardening reduction image.
 8. Themethod as claimed in claim 1, comprising generating one or more bonemarrow images, and diagnosing, identifying or monitoring bone marrowrelated disease using the one or more bone marrow images; generating oneor more knee cartilage images, and diagnosing, identifying or monitoringosteoarthritis or rheumatoid arthritis using the one or more bone marrowimages; generating one or more iodine contrast image, and diagnosing,identifying or monitoring a tumor; and/or generating one or more muscleimages, and diagnosing, identifying or monitoring sarcopenia.
 9. Themethod as claimed in claim 1, comprising generating any one or more (a)bone marrow images, (b) knee cartilage images, (c) iodine contrastimages, and (d) muscle images; generating one or more metal artefactimages and/or one or more beam hardening reduction images; and improvingimage quality of the bone marrow, knee cartilage, iodine contrast and/ormuscle images using the metal artefact and/or beam hardening reductionimages.
 10. A system for generating material decomposition images from aplurality of images obtained with plural-energy x-ray based imaging, theplurality of images corresponding to respective energies of theplural-energy x-ray based imaging, the system comprising: a neuralnetwork that has an encoder-decoder structure and includes a pluralityof encoder branches; wherein each of one or more of the plurality ofencoder branches is configured to encode two or more images of theplurality of images in combination; and each of one or more of theplurality of encoder branches is configured to encode a respectiveindividual image of the plurality of images; the neural network isconfigured to model spatial relationships and spectral relationshipsamong the plurality of images by learning features from the plurality ofimages in combination and one or more of the plurality of imagesindividually, and to generate one or more basis material imagesemploying the spatial relationships and the spectral relationships; andthe system is configured to generate one or more material specific ormaterial decomposition images from the basis material images.
 11. Thesystem as claimed in claim 10, wherein: i) each of two or more of theencoder branches is configured to encode a respective different image ofthe plurality of images; and/or ii) a first encoder branch is configuredto encode a first combination of two or more images of the plurality ofimages and a second encoder branch is configured to encode a secondcombination of two or more images of the plurality of images, whereinthe first combination is different from the second combination.
 12. Thesystem as claimed in claim 10, wherein the plural-energy x-ray basedimaging comprises cold cathode x-ray radiography, dual-energyradiography, multi-energy radiography, photon-counting radiography, coldcathode x-ray CT, dual-energy CT, multi-energy CT or photon-counting CT.13. The system as claimed in claim 10, wherein: a) the encoder branchesconfigured to encode a respective individual image encode in total allof the images that are encoded in total by the encoder branchesconfigured to encode two or more images; or b) the encoder branches thatencode a respective individual image are configured to encode in totalfewer images than are encoded in total by the encoder branches thatencode two or more images; or c) the encoder branches that encode arespective individual image are configured to encode in total moreimages than are encoded in total by the encoder branches that encode twoor more images.
 14. The system as claimed in claim 10, wherein the deeplearning neural network is a trained neural network, trained with realor simulated training images obtained with real or simulatedplural-energy x-ray based imaging and with basis material images. 15.The system as claimed in claim 14, wherein the basis material imagescomprise any one or more (i) hydroxyapatite images, (ii) calcium images,(iii) water images, (vi) fat images, (v) iodine images, and (iv) muscleimages.
 16. The system as claimed in claim 10, configured to generateany one or more of (i) a bone marrow decomposition image, (ii) a kneecartilage decomposition image, (iii) an iodine contrast decompositionimage, (iv) a tumor decomposition image, (v) a muscle and fatdecomposition image, (vi) a metal artefact reduction image, and (vii) abeam hardening reduction image.
 17. The system as claimed in claim 10,configured to generate one or more bone marrow images, and to diagnose,identify or monitor bone marrow related disease using the one or morebone marrow images; to generate one or more knee cartilage images, andto diagnose, identify or monitor osteoarthritis or rheumatoid arthritisusing the one or more bone marrow images; to generate one or more iodinecontrast image, and to diagnose, identify or monitor a tumor; and/or togenerate one or more muscle images, and to diagnose, identify or monitorsarcopenia.
 18. The system as claimed in claim 10, configured togenerate any one or more (a) bone marrow images, (b) knee cartilageimages, (c) iodine contrast images, and (d) muscle images; generate oneor more metal artefact images and/or one or more beam hardeningreduction images; and improve image quality of the bone marrow, kneecartilage, iodine contrast and/or muscle images using the metal artefactand/or beam hardening reduction images.
 19. A material decompositionimage, generated according to a method for generating materialdecomposition images from a plurality of images obtained withplural-energy x-ray based imaging, the plurality of images correspondingto respective energies of the plural-energy x-ray based imaging, themethod comprising: modelling spatial relationships and spectralrelationships among the plurality of images by learning features fromthe plurality of images in combination and one or more of the pluralityof images individually with a deep learning neural network; generatingone or more basis material images employing the spatial relationshipsand the spectral relationships; and generating one or more materialspecific or material decomposition images from the basis materialimages; wherein the neural network has an encoder-decoder structure andincludes a plurality of encoder branches; each of one or more of theplurality of encoder branches encodes two or more images of theplurality of images in combination; and each of one or more of theplurality of encoder branches encodes a respective individual image ofthe plurality of images.
 20. A non-transient computer-readable medium,comprising a computer program comprising program code configured, whenexecuted by one of more computing devices, to implement a method forgenerating material decomposition images from a plurality of imagesobtained with plural-energy x-ray based imaging, the plurality of imagescorresponding to respective energies of the plural-energy x-ray basedimaging, the method comprising: modelling spatial relationships andspectral relationships among the plurality of images by learningfeatures from the plurality of images in combination and one or more ofthe plurality of images individually with a deep learning neuralnetwork; generating one or more basis material images employing thespatial relationships and the spectral relationships; and generating oneor more material specific or material decomposition images from thebasis material images; wherein the neural network has an encoder-decoderstructure and includes a plurality of encoder branches; each of one ormore of the plurality of encoder branches encodes two or more images ofthe plurality of images in combination; and each of one or more of theplurality of encoder branches encodes a respective individual image ofthe plurality of images.